Display apparatus, image processing apparatus and image processing method, imaging apparatus, and program

ABSTRACT

A system and method displays a realistic image that allows a user to readily grasp his/her own positional relationship and also to experience a sense of presence. An outer dome screen is disposed so as to surround a user, and an immersion image such as the scenery of the surroundings of an object is displayed thereon. An inner dome screen is disposed inside the outer dome screen, and it displays a bird&#39;s-eye image of the object as perceived by the vision of the user when the object is viewed from a viewpoint of the user. In this case, the user is allowed to readily grasp his/her own positional relationship by the bird&#39;s-eye image and is allowed to experience a sense of presence by the immersion image.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to display apparatuses, image processingapparatuses as well as image processing methods, imaging apparatuses,and programs. Particularly, the present invention relates to a displayapparatus, an image processing apparatus, an image processing method, animaging apparatus, and a program that allow an image to be presented toa user with a greater sense of presence than conventional devices andmethods.

2. Discussion of the Background

As an example of a conventional apparatus for presenting a user with animage with a sense of presence, a display apparatus including animmersion screen is known. In the display apparatus including theimmersion screen, the user is presented with an image that would beperceived by the vision of the user assuming that the user were in avirtual space (hereinafter referred to as an immersion image whereappropriate). Thus, by viewing the immersion image, the user is able toexperience a sense of presence as if the user were actually present inthe virtual space provided by the immersion image.

The present inventors recognized several deficiencies with regard to theabove-described conventional systems and methods. For example, since thedisplay apparatus including the immersion screen, described above,displays an image that would be perceived by the vision of the userassuming that the user were in a virtual space, it is sometimesdifficult for the user to grasp a relative position of the user in thevirtual space provided by the immersion image.

As another example, in an automotive racing game (i.e., a racing videogame), a bird's-eye image of an automobile driven by a user is displayedas viewed from a certain viewpoint. In this case, the user is able toreadily grasp the position of the automobile the user is driving, andthus drive the automobile in a more realistic manner.

However, in a case where a bird's-eye image is displayed, compared witha case where an immersion image is displayed, a sense of immersion and asense of presence are diminished.

SUMMARY OF THE INVENTION

The present invention has been made in view of the above limitationswith conventional apparatuses and methods. The present invention enablesa user to readily grasp his/her own positional relationship and alsoallows an image with a sense of presence to be displayed.

A display apparatus according to the present invention includes a firstdisplay mechanism for displaying a first image, which is disposed so asto surround the user; and a second display mechanism for displaying asecond image, which is disposed inside the first display mechanism.

An image processing apparatus according to the present inventionincludes an image selecting mechanism for selecting a viewpoint image ofa predetermined object as viewed from a viewpoint of a user, from amongimages captured respectively from a plurality of positions, based on theviewpoint of the user; and a viewpoint-image converting mechanism forconverting the viewpoint image into a light-emission image based on theviewpoint of the user and a shape of the display mechanism.

An image processing method according to the present invention includesan image selecting step of selecting a viewpoint image of apredetermined object as viewed from a viewpoint of a user, from amongimages captured respectively from a plurality of positions, based on theviewpoint of the user; and a viewpoint-image converting step ofconverting the viewpoint image into a light-emission image based on theviewpoint of the user and a shape of display mechanism.

A program according to the present invention includes an image selectingstep of selecting a viewpoint image of a predetermined object as viewedfrom a viewpoint of a user, from among images captured respectively froma plurality of positions, based on the viewpoint of the user; and aviewpoint-image converting step of converting the viewpoint image into alight-emission image based on the viewpoint of the user and a shape ofdisplay mechanism.

An imaging apparatus according to the present invention includes anomnidirectional imaging mechanism for imaging an object's surroundingsin all directions, and a plurality of object-imaging mechanisms forimaging the object from a plurality of directions.

According to the display apparatus of the present invention, the firstdisplay mechanism that is disposed so as to surround (or at leastpartially surround, such as in a hemispherical context) a user displaysa first image, and the second display mechanism disposed inside (e.g.,within the hemispherical space) the first display mechanism displays asecond image.

According to the image processing apparatus, image processing method,and program of the present invention, a viewpoint image of apredetermined object as viewed from a viewpoint of a user is selectedbased on the viewpoint of the user, and the viewpoint image is convertedinto a light-emission image based on the viewpoint of the user and ashape of display mechanism.

According to the imaging apparatus of the present invention,surroundings in all directions around an object are imaged, and theobject is imaged from a plurality of directions.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1: Diagram showing an example configuration of an imaging/displaysystem according to an embodiment of the present invention.

FIG. 2: Block diagram showing an example configuration of an imagingsystem.

FIG. 3: Perspective view showing an example configuration of awide-angle imaging unit.

FIG. 4: Perspective view showing an example configuration of asurrounding imaging unit.

FIG. 5: Diagram illustrating a relationship of positions of anomnidirectional camera and surrounding cameras.

FIG. 6: Block diagram showing an example configuration of a displaysystem.

FIG. 7: Perspective view showing an example configuration of awide-angle-image display unit.

FIG. 8: Perspective view showing an example configuration of asurrounding-image display unit.

FIG. 9: Diagram illustrating the relationship of positions of an outerdome screen and an inner dome screen.

FIG. 10: Diagram illustrates a case where a plurality of users areviewing images.

FIG. 11: Diagram showing a case where images of an object as viewed fromrespective viewpoints of a plurality of users are presented to theusers.

FIG. 12: Diagram showing a horizontal parallax system.

FIG. 13: Diagram showing a vertical-and-horizontal parallax system.

FIG. 14: Block diagram showing an example configuration of awide-angle-image processing unit.

FIG. 15: Diagram showing a wide-angle-imaging coordinate system.

FIG. 16: Diagram showing an omnidirectional image.

FIG. 17: Diagram showing an omnidirectional image and alongitude-latitude image.

FIG. 18: Flowchart for explaining wide-angle-image processing.

FIG. 19: Diagram for explaining an image that is displayed on an innerdome screen.

FIG. 20: Diagram showing an image that is displayed on a display surfaceof an inner omni-directional projector and an image that is displayed onthe inner dome screen.

FIG. 21: Diagram showing a case where viewpoints of users A and B areremote from each other.

FIG. 22: Diagram showing a case where viewpoints of users A and B areclose to each other.

FIG. 23: Block diagram showing an example configuration of asurrounding-image processing unit.

FIG. 24: Diagram for explaining processing that is executed by aviewpoint-direction calculating unit.

FIG. 25: Diagram for explaining processing that is executed by adisplay-image generating unit.

FIG. 26: Diagram for explaining processing that is executed by thedisplay-image generating unit.

FIG. 27: Diagram for explaining processing that is executed by thedisplay-image generating unit.

FIG. 28: Flowchart for explaining surrounding-image processing.

FIG. 29: Flowchart for explaining the surrounding-image processing.

FIG. 30: Block diagram showing an example configuration of a computeraccording to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 shows an example configuration of an imaging/display system (asystem refers to a logical combination of a plurality of apparatuses,regardless of whether the apparatuses are in the same enclosure)according to an embodiment of the present invention.

An imaging system 1 captures images of an object and images in alldirections around (except, perhaps, beneath) the object. The imagingsystem 1 records the image data obtained on a recording medium 3, forexample, an optical disk, a magnetic disk, a magnetic tape, or asemiconductor memory, or sends the image data via a transmission medium4, for example, a telephone circuit, optical fiber, a satellite circuit,a microwave or other wireless link, a CATV (cable television) network,or the Internet.

A display system 2 plays back image data from the recording medium 3, orreceives and displays image data transmitted via the transmission medium4.

FIG. 2 shows an example configuration of the imaging system 1 shown inFIG. 1. A wide-angle imaging unit 11 includes a wide-angle camera (videocamera) for imaging the surroundings of the object over a wide angle,and it supplies wide-angle-image data (video data) captured by thewide-angle camera to a multiplexer 13. The wide-angle cameraconstituting the wide-angle imaging unit 11 is implemented, for example,by an omnidirectional camera for capturing images in all directions (orsubstantially all directions, such as all directions except directlybeneath the object, or even all directions with a field of view, orstill all directions with a substantially hemispherical volume aroundthe object) around the object, as will be described later. Thus, thewide-angle image data output from the wide-angle imaging unit 11includes images such as the scenery of the surroundings of the object.

A surrounding imaging unit 12 includes a large number of cameras (videocameras) disposed so as to surround the object in horizontal andvertical directions, and it supplies image data (video data) captured bythe plurality of cameras to the multiplexer 13. In the surroundingimaging unit 12, the cameras disposed at a plurality of positionsrespectively output image data obtained by imaging the object from thepositions of the cameras (camera positions). The image data of theobject, captured from the plurality of positions, will be collectivelyreferred to as surrounding-image data when appropriate.

The multiplexer 13 multiplexes the wide-angle-image data supplied fromthe wide-angle imaging unit 11 and the surrounding-image data suppliedfrom the surrounding imaging unit 12, and supplies the resultingmultiplexed image data to a recording/sending unit 14.

More specifically, the multiplexer 13 multiplexes wide-angle-image dataand surrounding-image data of the same frame in association with eachother. Also, the multiplexer 13 multiplexes each frame ofsurrounding-image data, i.e., image data captured from the plurality ofcamera positions, in association with the camera positions. It is to beassumed that the camera positions of the respective cameras constitutingthe surrounding imaging unit 12 are known in advance. The camerapositions are represented, for example, by coordinates representingpositions on a three-dimensional coordinate system having an origin atthe position of the object.

The recording/sending unit 14 records the multiplexed image datasupplied from the multiplexer 13 on the recording medium 3 (FIG. 1), orsends the multiplexed image data via the transmission medium 4.

FIG. 3 shows an example configuration of the wide-angle imaging unit 11shown in FIG. 2. The wide-angle imaging unit 11 includes anomnidirectional camera 21. Ideally, the omnidirectional camera 21receives light beams that are incident on the omnidirectional camera 21through a hemispherical surface as shown in FIG. 3, and outputs imagedata corresponding to the light beams.

The omnidirectional camera 21 is implemented, for example, by a camerawith a fish-eye lens, or a camera with a hyperboloid mirror. In thisembodiment, the omnidirectional camera 21 is implemented by a camerawith a fish-eye lens. The resultant image of the object is captured insubstantially all directions with respect to an observation point, whichin this case is the location of the omnidirectional camera.

The omnidirectional camera 21 is disposed so as to allow imaging of thescenery, etc. in all directions around the object. The use of the term“all directions” is used herein in a general sense and is not intendedto be narrowly interpreted; for instance the direction immediatelybeneath the camera or in the shadow of the object may not be included.For example, if the object is a soccer match that is being played in astadium, the omnidirectional camera 21 is disposed, for example, at acenter of the stadium, so that the omnidirectional camera 21 capturesimages of the scenery of the stand where spectators are watching thesoccer match, or even the sky, or inside of the stadium dome.

FIG. 4 shows an example configuration of the surrounding imaging unit 12shown in FIG. 2. The surrounding imaging unit 12 includes a large numberof surrounding cameras 22 _(k) (k=1, 2, . . . ) that are disposed aroundthe object so as to surround the object, so that the surrounding imagingunit 12 captures images of the object as viewed from a large number ofviewpoints. Thus, for example, if the object is a soccer match that isbeing played in a stadium as described earlier, the large number ofsurrounding cameras 22 _(k) constituting the surrounding imaging unit 12are respectively disposed so as to surround (i.e., at least within thegamut of candidate venues for watching the soccer field) a soccer fieldprovided in the stadium, so that the surrounding cameras 22 _(k)captures images of what is going on in the soccer match on the soccerfield.

The surrounding cameras 22 _(k) may be disposed in a horizontal planeonly in longitudinal directions (horizontal directions) of the object orin all longitudinal and latitudinal directions (horizontal and verticaldirections) of the object. If the surrounding cameras 22 _(k) aredisposed only in the longitudinal directions of the object in ahorizontal plane, only a horizontal parallax system, which will bedescribed later, can be employed. If the surrounding cameras 22 _(k) aredisposed in all longitudinal and latitudinal directions of the object,either a horizontal parallax system or a vertical-and-horizontalparallax system, which will be described later, can be employed.

FIG. 5 shows the relationship of positions where the omnidirectionalcamera 21 (FIG. 3) and the surrounding cameras 22 _(k) (FIG. 4) aredisposed. The surrounding cameras 22 _(k) are disposed, for example, ona spherical surface of a hemisphere surrounding the object, at regularlongitudinal and latitudinal intervals. The surrounding cameras 22 _(k)image the object with the positions of the surrounding cameras 22 _(k)(camera positions) as viewpoints.

The omnidirectional camera 21 is disposed, for example, at a center ofthe hemisphere where the surrounding cameras 22 _(k) are disposed. Theomnidirectional camera 21 images the surroundings of the object in alldirections, i.e., in directions of points located on the curved surfaceof the hemisphere where the surrounding cameras 22 _(k) are disposedfrom the center of the hemisphere. Moreover, the term “in alldirections” refers to directions that correspond to points on a surfacefrom which there are candidate vistas of at least a portion of a spacein which an object is located. This surface may also include the planarsurface portion of the hemispherical shape, but need not (although may)include directions that are outside the field of view of theomnidirectional camera 21.

In the embodiment shown in FIG. 5, the omnidirectional camera 21 ispositioned so that an optical axis thereof extends in the direction of anorth pole of the hemisphere where the surrounding cameras 22 _(k) aredisposed (in the direction of a point with latitude of 90 degrees). Thesurrounding cameras 22 _(k) are disposed so that optical axes thereofextend in the direction of the center of the hemisphere.

It is to be understood that the omnidirectional camera 21 and thesurrounding cameras 22 _(k) may be disposed at positions other than thepositions shown in FIG. 5, or even in spaces other than in ahemispherical space, such as a cube, or other traditional ornon-traditionally defined 3 dimensional interior space (e.g., it may bean arbitrarily shaped space).

In the imaging system 1 shown in FIG. 2, in the wide-angle imaging unit11, the omnidirectional camera 21 (FIG. 3) captures images in alldirections around the object, and supplies obtainedomnidirectional-image data (wide-angle-image data) to the multiplexer13. At the same time, in the surrounding imaging unit 12, thesurrounding cameras 22 _(k) (FIG. 4) capture images of the object asviewed from the camera positions thereof, and supplies surrounding-imagedata obtained to the multiplexer 13.

The multiplexer 13 multiplexes the omnidirectional-image data suppliedfrom the wide-angle imaging unit 11 and the surrounding-image datasupplied from the surrounding imaging unit 12, and supplies theresulting multiplexed image data to the recording/sending unit 14.

The recording/sending unit 14 records the multiplexed image datasupplied from the multiplexer 13 on the recording medium 3 (FIG. 1), orsends the multiplexed image data to a destination (such as a computer ata remote location) via the transmission medium 4.

Thus, in the imaging system 1, for example, if the object is a soccermatch that is being played in a stadium, the scenery of the stands wherespectators are watching the soccer match, and a soccer ball kicked uphigh, etc. are imaged to obtain omnidirectional-image data, and imagesof the ground and players in the soccer match as viewed from a largenumber of viewpoints are captured to obtain surrounding-image data.

Next, FIG. 6 shows an example configuration of the display system 2shown in FIG. 1. A playback/receiving unit 31 plays back multiplexedimage data from the recording medium 3 (FIG. 1), or receives multiplexedimage data transmitted via the transmission medium 4 (FIG. 1), andsupplies the multiplexed image data to a demultiplexer 32.

The demultiplexer 32 demultiplexes the multiplexed image data suppliedfrom the playback/receiving unit 31 into frame-by-frameomnidirectional-image data and surrounding-image data, and supplies theomnidirectional-image data to a wide-angle-image processing unit 33while supplying the surrounding-image data to a surrounding-imageprocessing unit 35.

The wide-angle-image processing unit 33 processes theomnidirectional-image data supplied from the demultiplexer 32, andsupplies the resulting image data to a wide-angle-image display unit 34.

The wide-angle-image display unit 34, based on an image corresponding tothe image data supplied from the wide-angle-image processing unit 33,displays an immersion image that presents a user with a sense ofimmersion.

The surrounding-image processing unit 35 processes the surrounding-imagedata supplied from the demultiplexer 32 in accordance with a user'sviewpoint supplied from a viewpoint detecting unit 37, therebygenerating image data of the object as viewed from the user's viewpoint,and supplies the image data to a surrounding-image display unit 36.

The surrounding-image display unit 36, based on the image data suppliedfrom the surrounding-image processing unit 35, displays a bird's-eyeimage of the object as viewed from a particular viewpoint such as theuser's viewpoint.

The viewpoint detecting unit 37 detects a viewpoint of a user, andsupplies the user's viewpoint to the surrounding-image processing unit35. The viewpoint detecting unit 37 is implemented, for example, by amagnetic sensor from Polhemus, Inc., which functions as athree-dimensional position sensor. In that case, the viewpoint detectingunit 37 implemented by the magnetic sensor is mounted, for example, onthe head of the user. Alternatively, the viewpoint detecting unit 37 maybe implemented by a mechanism that allow user-controlled input regarding(change of) a viewpoint, such as a joystick or a track ball. That is,the viewpoint detecting unit 37 need not allow detection of an actualviewpoint of a user, and may be a mechanism that allows input of avirtual viewpoint of a user.

FIG. 7 shows an example configuration of the wide-angle-image displayunit 34 shown in FIG. 6. The wide-angle-image display unit 34 includesan outer omnidirectional projector 41 and an outer dome screen 42. Theouter omnidirectional projector 41 is capable of emitting light beamsover a wide angle, and is implemented, for example, by an ordinaryprojector and a fish-eye lens serving as an optical system that spreadslight beams emitted by the projector over a wide angle.

The outer dome screen 42 has a shape of, for example, a hemisphericaldome, and is disposed so as to surround the user. Furthermore, thehemispherical dome serving as the outer dome screen 42 is large enoughto allow a plurality of users to be accommodated therein and to movearound to a certain extent, and the diameter thereof is, for example, onthe order of 5 m.

On an inner (or outer) surface of the outer dome screen 42, for example,a coating material that causes light to be scattered is applied. Insidethe outer dome screen 42, the outer omnidirectional projector 41 isdisposed so that light beams will be emitted towards the inner surfaceof the outer dome screen 42, which in this hemispherical spaceconstitutes “all directions”.

More specifically, for example, the outer omnidirectional projector 41is disposed so that an optical center thereof coincides with a center ofthe hemispherical dome serving as the outer dome screen 42 and so thatan optical axis thereof is perpendicular to a horizontal plane. Theouter omnidirectional projector 41, using the fish-eye lens thereof,emits light beams corresponding to the image data supplied from thewide-angle-image processing unit 33 towards the entire inner surface ofthe outer dome screen 42 having a shape of a hemispherical dome.

The light beams emitted by the outer omnidirectional projector 41 arereceived and scattered at points on the inner surface of the outer domescreen 42 having a shape of a hemispherical dome. Thus, on the innersurface of the outer dome screen 42 disposed over the user, an immersionimage corresponding to the light beams emitted by the outeromnidirectional projector 41, i.e., the omnidirectional image of theobject captured by the imaging system 1, is displayed.

The shape of the outer dome screen 42 is not limited to a hemisphericaldome shape as long as the shape is such that the user is surrounded, andmay be, for example, a cylinder shape, a cube shape, or another3-dimensional shape that does not have a commonly recognized name.

Next, FIG. 8 shows an example configuration of the surrounding-imagedisplay unit 36 shown in FIG. 6. The surrounding-image display unit 35includes an inner omnidirectional projector 43 and an inner dome screen44.

The inner omnidirectional projector 43, similar to the outeromnidirectional projector 41 shown in FIG. 7, is capable of emittinglight beams over a wide angle, and is implemented, for example, by anordinary projector and a fish-eye lens serving as an optical system thatspreads light beams emitted by the projector over a wide angle.

The inner dome screen 44 has a predetermined shape of, for example, ahemispherical dome. On an inner (or outer) surface of the inner domescreen 44, for example, a coating material that causes light to bescattered is applied. Inside the inner dome screen 44, the inneromnidirectional projector 43 is disposed so that light beams will beemitted towards the inner surface of the inner dome screen 44.

More specifically, for example, the inner omnidirectional projector 43is disposed so that an optical center thereof coincides with a center ofthe hemispherical dome serving as the inner dome screen 44 and so thatan optical axis thereof is perpendicular to a horizontal plane. Theinner omnidirectional projector 43, using the fish-eye lens thereof,emits light beams corresponding to the image data supplied from thewide-angle-image processing unit 33 towards the entire inner surface ofthe inner dome screen 44 having a shape of a hemispherical dome.

The light beams emitted by the inner omnidirectional projector 43 arereceived and scattered at points on the inner surface of the inner domescreen 44 having a shape of a hemispherical dome. The inner dome screen44 itself is composed of a transparent material. The light beams emittedby the inner omnidirectional projector 43 are scattered on the innersurface of the inner dome screen 44, whereby a bird's eye imagecorresponding to the light beams emitted by the inner omnidirectionalprojector 43, i.e., an image of the object as viewed from a certainviewpoint, captured by the imaging system 1, is displayed on the outersurface of the inner dome screen 44 by what is called back projection.

Although the shape of the inner dome screen 44 in this example is ahemispherical dome, which is similar to the shape of the outer domescreen 42 shown in FIG. 7, the shape of the inner dome screen 44 neednot be similar to the shape of the outer dome screen 42. That is, theshape of the inner dome screen 44 may be, for example, a cylindricalshape, a polygonal-prism shape, or another shape that does not have acommonly recognizable name, independent of the shape of the outer domescreen 42.

FIG. 9 shows the relationship of positions where the outer dome screen42 (FIG. 7) and the inner dome screen 44 are disposed. As describedearlier, the outer dome screen 42 is disposed so as to surround theuser, and the inner dome screen 44 is disposed inside the outer domescreen 42. The use of the term “surround” in this context means that theouter dome screen 42 need not necessarily encapsulate the user, butmerely give the user a sense of being contained within the space definedby the outer dome screen 42. For example, the outer dome screen 42preferable rests on the same surface that the user is standing on.However, the outer dome screen 42 may also be suspended, or supported inanother way so that the outer dome screen 42 is elevated by some extent(e.g., an inch or two, up to a few feet).

Thus, the user in a region inside the outer dome screen 42 and outsidethe inner dome screen 44 is allowed to simultaneously view the immersionimage displayed on the inner surface of the outer dome screen 42 and thebird's-eye image displayed on the outer surface of the inner dome screen44. Accordingly, the user is allowed to experience a sense of immersionprovided by the immersion image and to readily grasp his/her ownposition by the bird's-eye image.

More specifically, for example, an immersion image of the scenery of thestand of the soccer stadium is displayed on the inner surface of theouter dome screen 42, and what is going on in the match on the soccerfield as viewed from a viewpoint of the user is displayed on the outersurface of the inner dome screen 44. Accordingly, the user is allowed toexperience a sense of presence as if the user is actually watching thesoccer match in the soccer stadium.

As shown in FIG. 10, not only one, but a plurality of users can beaccommodated inside the outer dome screen 42. Thus, with the displaysystem 2, a plurality of users is allowed to simultaneously view theimmersion image displayed on the outer dome screen 42 and the bird's-eyeimage displayed on the inner dome screen 44.

For example, if two users A and B are present inside the outer domescreen 42 and if the users A and B are at different positions as shownin FIG. 11, the inner dome screen 44 presents the user A with abird's-eye image of the object as viewed from a viewpoint of the user Awhile presenting the user B with a bird's-eye image of the object asviewed from a viewpoint of the user A.

That is, in the display system 2, the inner dome screen 44 presents eachuser with a bird's-eye image of the object as viewed from a viewpoint ofthe user.

Thus, the user is allowed to view an image (bird's-eye image) of theobject as viewed from his/her viewpoint, and has the freedom forselecting a viewpoint. Viewpoint positions that can be selected by theuser may be, for example, limited to positions in a horizontal plane ata certain height inside the outer dome screen 42, as shown in FIG. 12,or may be, for example, any positions inside the outer dome screen 42without such a limitation, as shown in FIG. 13. The system in whichviewpoints are limited to positions on a certain horizontal plane, asshown in FIG. 12, can be called a horizontal parallax system since theuser is allowed to move his/her viewpoint only in horizontal directions.The system in which viewpoints may be at any position, as shown in FIG.13, can be called a vertical-and-horizontal parallax system since theuser is allowed to move his/her viewpoint in any vertical and horizontaldirection.

FIG. 14 shows an example configuration of the wide-angle-imageprocessing unit 33 shown in FIG. 6. The frame-by-frameomnidirectional-image data output from the demultiplexer 32 is suppliedto a frame memory 51. The frame memory 51 stores, on a frame-by-framebasis, the omnidirectional-image data supplied from the demultiplexer32, and reads the omnidirectional image data on a frame-by-frame basisand supplies it to a wide-angle-image converting unit 52. The framememory 51 includes a plurality of memory banks, and it allowssimultaneous storing and reading of omnidirectional-image data byswitching of the banks.

The wide-angle-image converting unit 52, under the control of acontroller 56, converts the omnidirectional-image data supplied from theframe memory 51 into latitude-longitude-image data representing arectangular region on a plane defined by a latitudinal direction and alongitudinal direction, and supplies it to an image-angle correctingunit 53.

The image-angle correcting unit 53, under the control of the controller56, corrects the latitude-longitude-image data supplied from thewide-angle-image converting unit 52, based on optical characteristics ofthe outer omnidirectional projector 41 of wide-angle-image display unit34 (FIG. 7), and supplies the latitude-longitude-image data having beencorrected to a latitude-longitude-image converting unit 54.

The latitude-longitude-image converting unit 54, under the control ofthe controller 56, converts the latitude-longitude-image data suppliedfrom the image-angle correcting unit 53 into omnidirectional-image data,and supplies it to a frame memory 55.

The frame memory 55 stores, on a frame-by-frame basis, theomnidirectional-image data supplied from the latitude-longitude-imageconverting unit 54, and reads the omnidirectional-image data on aframe-by-frame basis and supplies it to the outer omnidirectionalprojector 41 of the wide-angle-image display unit 34 (FIG. 7). The framememory 55, similarly to the frame memory 51, includes a plurality ofmemory banks, and it allows simultaneous storing and reading ofomnidirectional-image data by switching of the banks.

The controller 56, based on the status of storage ofomnidirectional-image data in the frame memory 51, controls thewide-angle-image converting unit 52, the image-angle correcting unit 53,and the latitude-longitude-image converting unit 54.

Next, processing that is executed by the wide-angle-image convertingunit 52, the image-angle correcting unit 53, and thelatitude-longitude-image converting unit 54 shown in FIG. 14 will bedescribed with reference to FIGS. 15 to 17.

The wide-angle-image converting unit 52 receives omnidirectional-imagedata stored in the frame memory 51, i.e., omnidirectional-image datacaptured by the omnidirectional camera 21 constituting the wide-angleimaging unit 11 shown in FIG. 3.

Now, letting a three-dimensional coordinate system shown in FIG. 15, inwhich an optical axis of the omnidirectional camera 21 constitutes a zaxis and in which an imaging surface (photoreceptor surface) of theomnidirectional camera 21 constitutes an xy plane, be referred to as awide-angle-imaging coordinate system, the relationship between lightbeams that are incident on the omnidirectional camera 21 andomnidirectional-image data captured by the omnidirectional camera 21receiving the light beams will be considered in the wide-angle-imagingcoordinate system.

The light beams L emitted toward the omnidirectional camera 21 arerefracted by the fish-eye lens thereof and becomes incident on theimaging surface of the omnidirectional camera 21. That is, the fish-eyelens refracts the light beams L in a plane including the Z axis of thewide-angle-imaging coordinate system and the light beams L so that thelight beams L will be incident on the xy plane. Thus, theomnidirectional camera 21 captures, as omnidirectional-image data, imagedata representing, for example, a circular region or a doughnut-shapedregion shown as hatched in FIG. 16.

Now, with regard to the wide-angle-imaging coordinate system, let anangle with respect to the x axis on the xy plane be referred to as alongitude θ, an angle with respect to the z axis be referred to as alatitude φ and the direction of a point with the longitude θ and thelatitude φ be denoted as a direction (θ, φ), as shown in FIG. 15. Then,a light beam L that is incident on the omnidirectional camera 21 fromthe direction (θ, φ) is refracted by the fish-eye lens of theomnidirectional camera 21 in accordance with the latitude φ.

Let the light beam L that is incident from the direction (θ, φ) bedenoted as a light beam L(θ, φ), and a point on the xy plane to whichthe light beam L(θ, φ) is projected as Q(x, y). Then, the longitude θand the latitude φ can be expressed by the following equations using thex and y coordinates of the point Q.

$\begin{matrix}{\theta = {\tan^{- 1}\frac{y}{x}}} & (1)\end{matrix}$φ=f ₁(√{square root over (x ² +y ²)})  (2)

In equation (2), the function f₁( ) is determined according to opticalcharacteristics of the fish-eye lens of the omnidirectional camera 21.Letting the inverse function of the function f₁( ) be denoted as f₁ ⁻¹ (), equation (2) can be rewritten as √(x²+y²)=f₁ ⁻¹ (θ). A fish-eye lensfor which the inverse function f₁ ⁻¹ (θ), representing the opticalcharacteristics of the fish-eye lens, can be expressed as (the focallength of the fish-eye lens)×θ is called an equidistance projectionmethod.

From equations (1) and (2), a point Q(x, y) of the omnidirectional imagecaptured by the omnidirectional camera 21 can be expressed using alongitude θ and a latitude φ representing a direction of a light beam L(θ, φ) that is incident on that point.

More specifically, an angle that is formed by a line connecting thepoint Q(x, y) and the origin O with respect to the x axis is equal tothe longitude θ of the light beam L. The distance between the point Q(x,y) and the origin O, √(x²+y²), depends on the latitude φ of the lightbeam L. The longitude θ and the latitude φ are values independent ofeach other.

Thus, for example, assuming a latitude-longitude coordinate system thatis a two-dimensional coordinate system with the longitude θ on thehorizontal axis and the latitude φ, on the vertical axis, a pixel Q(x,y) constituting the omnidirectional-image data can be projected to apoint Q′(θ, φ) on the latitude-longitude coordinate system.

By projecting the omnidirectional-image data onto the latitude-longitudecoordinate system, the omnidirectional-image data representing adoughnut-shaped region is converted into image data representing arectangular region defined by the longitude θ and the latitude φ, i.e.,latitude-longitude-image data, as shown in FIG. 17.

The wide-angle-image converting unit 52 shown in FIG. 14 convertsomnidirectional-image data into latitude-longitude-image data in themanner described above.

If the optical characteristics of the fish-eye lens of theomnidirectional camera 21 of the wide-angle imaging unit 11 (FIG. 3) forcapturing an omnidirectional image are the same as the opticalcharacteristics of the fish-eye lens of the outer omnidirectionalprojector 41 of the wide-angle-image display unit 34 (FIG. 7) foremitting light beams corresponding to the omnidirectional image, whenomnidirectional-image data captured by the omnidirectional camera 21 issupplied to the outer omnidirectional projector 41 as it is, the outeromnidirectional projector 41 emits light beams that are different fromlight beams that are incident on the omnidirectional camera 21 only inthat the directions are opposite. Thus, theoretically, the outer domescreen 42 (FIG. 7), which receives the light beams from the outeromnidirectional projector 41, displays the same image of the scenery,etc. as captured by the omnidirectional camera 21.

However, the optical characteristics of the fish-eye lenses of theomnidirectional camera 21 (FIG. 3) and the outer omnidirectionalprojector 41 (FIG. 7) are not necessarily the same and may differ fromeach other.

The point Q(x, y) on the imaging surface that a light beam L(θ, φ)incident on the omnidirectional camera 21 reaches is related to thelongitude θ and the latitude φ as expressed in equations (1) and (2).

The longitude θ expressed by equation (1) is not affected by the opticalcharacteristics of the fish-eye lens. On the other hand, the latitude φexpressed in equation (2) is calculated using the function f₁ ( ), andsince the inverse function f₁ ⁻¹ ( ) of the function f₁ ( ) representsthe optical characteristics of the fish-eye lens as described earlier,the latitude φ is affected by the optical characteristics of thefish-eye lens.

Thus, when the optical characteristics of the fish-eye lenses of theomnidirectional camera 21 (FIG. 3) and the outer omnidirectionalprojector 41 (FIG. 7) differ from each other, in order that the imagedisplayed on the outer dome screen 42 (FIG. 7) be the same as the imageof the scenery, etc. as captured by the omnidirectional camera 21, theomnidirectional-image data captured by the omnidirectional camera 21must be corrected with respect to the direction of the latitude φ.

Since the direction of the latitude φ of the omnidirectional-image datais along the direction of the vertical axis of thelatitude-longitude-image data shown in FIG. 17, the image-anglecorrecting unit 53 shown in FIG. 14 corrects the latitude φ on thevertical axis of the latitude-longitude-image data supplied from thewide-angle-image converting unit 52 into a latitude φ′, for example,according to the following equation.φ′=αφ  (3)

In equation (3), α is a value that is determined by the opticalcharacteristics of the fish-eye lenses of the omnidirectional camera 21(FIG. 3) and the outer omnidirectional projector 41 (FIG. 7).

Thus, letting a pixel constituting the latitude-longitude-image datasupplied from the wide-angle-image converting unit 52 be denoted asQ′(θ, φ), the image-angle correcting unit 53 corrects the pixel Q′(θ, φ)into a pixel Q′(θ, φ′).

The latitude φ, which represents the direction of the vertical axis ofthe latitude-longitude-image data as shown in FIG. 17, corresponds to animage angle in the vertical direction of the latitude-longitude-imagedata. Thus, the correction of the latitude φ corresponds to increasingor decreasing the image angle in the vertical direction of thelatitude-longitude-image data.

If the latitude-longitude-image data representing a rectangular regionis supplied as it is to the outer omnidirectional projector 41 (FIG. 7),the outer dome screen 42 does not display the omnidirectional imagecaptured by the omnidirectional camera 21 (FIG. 3). Thus, thelatitude-longitude-image converting unit 54 shown in FIG. 14 convertsthe latitude-longitude-image data with the latitude φ thereof havingbeen corrected in the image-angle correcting unit, back intoomnidirectional-image data representing a doughnut-shaped region asshown in FIG. 16 (or a circular region). In the omnidirectional-imagedata yielded by the latitude-longitude-image converting unit 54, thescale along the direction of the latitude φ has been corrected based onthe optical characteristics of the fish-eye lenses of theomnidirectional camera 21 (FIG. 3) and the outer omnidirectionalprojector 41 (FIG. 7). Thus, when the omnidirectional-image data issupplied to the outer omnidirectional projector 41 (FIG. 7), the outerdome screen 42 displays the omnidirectional image captured by theomnidirectional camera 21 (FIG. 3).

Next, processing that is executed in the wide-angle-image processingunit 33 shown in FIG. 14 (wide-angle-image processing) will be describedwith reference to a flowchart shown in FIG. 18.

The controller 56 monitors the frame memory 51, and it startswide-angle-image processing when the frame memory 51 starts storingomnidirectional-image data.

More specifically, in the wide-angle-image processing, first in step S1,the controller 56 sets an initial value, for example, 1, in a variable ifor counting the number of frames, and the processing then proceeds tostep S2. In step S2, the frame memory 51 reads omnidirectional-imagedata of an i-th frame stored therein, and supplies it to thewide-angle-image converting unit 52. The processing then proceeds tostep S3.

In step S3, the wide-angle-image converting unit 52 calculates,according to equation (1), a longitude θ for each pixel Q(x, y)constituting the omnidirectional-image data supplied from the framememory 51, and the processing then proceeds to step S4. In step S4, thewide-angle-image converting unit 52 calculates, according to equation(2), a latitude φ for each pixel Q(x, y) constituting theomnidirectional-image data supplied from the frame memory 51, and theprocessing then proceeds to step S5.

In step S5, the wide-angle-image converting unit 52 maps each pixel Q(x,y) constituting the omnidirectional-image data supplied from the framememory 51 to a point (θ, φ) on the latitude-longitude coordinate system,represented by the longitude θ and the latitude φ that have beencalculated for the pixel Q(x, y), thereby converting theomnidirectional-image data representing a doughnut-shaped region intolatitude-longitude-image data representing a rectangular region, asshown in FIG. 17.

The wide-angle-image converting unit 52, having obtained thelatitude-longitude-image data in step S5 as described above, suppliesthe latitude-longitude-image data to the image-angle correcting unit 53,and the processing then proceeds to step S6.

In step S6, the image-angle correcting unit 53 corrects the latitude φof the latitude-longitude-image data supplied from the wide-angle-imageconverting unit 52 into a latitude φ′ according to equation (3), andsupplies latitude-longitude-image data represented by the correctedlatitude φ′ and the longitude θ to the latitude-longitude-imageconverting unit 54. The processing then proceeds to step S7.

In step S7, the latitude-longitude-image converting unit 54 calculates,according to equation (4) (see below), an x coordinate on the xy planeof the wide-angle-imaging coordinate system, for each pixel Q′(θ, φ′)constituting the longitude-latitude-image data supplied from theimage-angle correcting unit 53, and the processing then proceeds to stepS8.

In step S8, the latitude-longitude-image converting unit 54 calculates,according to equation (5), a y coordinate on the xy plane of thewide-angle-imaging coordinate system, for each pixel Q′(θ, φ′)constituting the longitude-latitude-image data supplied from theimage-angle correcting unit 53, and the processing then proceeds to stepS9.x=f ₂(φ′)cos θy=f ₂(φ′)sin θ  (4)

The function f₂(φ′) in equations (4) and (5) represents the opticalcharacteristics of the outer omnidirectional projector 41 (FIG. 7) (afunction that is determined by a projection method of the fish-eye lensof the outer omnidirectional projector 41). Solving equations (1) and(2) yields equations similar to equations (4) and (5).

In step S9, the latitude-longitude-image converting unit 54 maps eachpixel Q′(θ, φ′) constituting the longitude-latitude-image data suppliedfrom the image-angle correcting unit 53 to a point (x, y) on the xyplane of the wide-angle-imaging coordinate system, represented by the xand y coordinates obtained for the pixel Q′(θ, φ′), thereby convertingthe latitude-longitude-image data representing a rectangular region intoomnidirectional-image data representing a doughnut-shaped region. Theomnidirectional-image data is supplied to and stored in the frame memory55, as light-emission omnidirectional-image data for causing the outeromnidirectional projector 41 (FIG. 7) to emit light.

The light-emission omnidirectional-image data stored in the frame memory55 as described above is sequentially read and supplied to thewide-angle-image display unit 34 (FIG. 6). In the wide-angle-imagedisplay unit 34, the outer omnidirectional projector 41 (FIG. 7) emitslight beams corresponding to the light-emission omnidirectional-imagedata, and the outer dome screen 42 (FIG. 7) receives the light beams,whereby the outer dome screen 42 displays, as an immersion image, animage of the scenery in all directions around the object, captured bythe omnidirectional camera 21 (FIG. 3).

Then, the processing proceeds to step S10, in which the controller 51refers to the frame memory 51 to determine whether the i-th frame is thelast frame. If it is determined in step S10 that the i-th frame is notthe last frame, that is, if omnidirectional-image data of a framesubsequent to the i-th frame is stored in the frame memory 51, theprocessing proceeds to step S11, in which the controller 56 incrementsthe variable i by 1. The processing then returns to step S2, and similarthe same processing is repeated.

On the other hand, if it is determined in step S10 that the i-th frameis the last frame, that is, if omnidirectional-image data of a framesubsequent to the i-th frame is not stored in the frame memory 51, thewide-angle-image process is stopped.

If the omnidirectional camera 21 (FIG. 3) and the outer omnidirectionalprojector 41 (FIG. 7) have the same optical characteristics,omnidirectional-image data output by the demultiplexer 32 (FIG. 6) maybe directly supplied to the wide-angle-image display unit 34 aslight-emission omnidirectional-image data, bypassing thewide-angle-image processing unit 33 shown in FIG. 14.

Next, an overview of the surrounding-image processing unit 35 shown inFIG. 6 will be described. In the surrounding-image display unit 36 (FIG.6), the inner dome screen 44 (FIG. 8) presents the user with an image ofthe object as viewed from a viewpoint of the user, as described earlier.If a plurality of users exists, the inner dome screen 44 (FIG. 8)presents the plurality of users with images of the object as viewed fromthe respective viewpoints of the users.

Thus, if the inner dome screen 44 is transparent and if the objectimaged by the surrounding imaging unit 12 (FIG. 2) exists inside theinner dome screen 44, the inner dome screen 44 displays the same imageof the object as perceived by the vision of each user, as shown in FIG.19.

More specifically, in the embodiment shown in FIG. 19, two users A and Bare present, and the inner dome screen 44 displays an image of theobject as viewed from a viewpoint of the user A at a point where astraight line connecting the center of the inner dome screen 44 and theviewpoint of the user A intersects the inner dome screen 44.Furthermore, the inner dome screen 44 displays an image of the object asviewed from a viewpoint of the user B at a point where a straight lineconnecting the center of the inner dome screen 44 and the viewpoint ofthe user B intersects the inner dome screen 44.

Thus, in the surrounding display unit 36 shown in FIG. 8, for example,as shown in FIG. 20, the inner omnidirectional projector 43 displays animage of the object as viewed from a viewpoint of each user (hereinafterreferred to as a viewpoint image when appropriate) on a display surfaceof, for example, a liquid crystal panel that is not shown, and emitslight beams corresponding to the viewpoint image through the fish-eyelens. Accordingly, the inner dome screen 44 presents each user with animage of the object as viewed from a viewpoint of the user.

The surrounding-image processing unit 35 shown in FIG. 6, using thesurrounding-image data supplied from the demultiplexer 32, generates animage of the object as viewed from each viewpoint supplied from theviewpoint detecting unit 37 (viewpoint image), and maps the viewpointimage associated with each viewpoint to a position in accordance withthe viewpoint, thereby generating a light-emission image correspondingto light beams to be emitted by the inner omnidirectional projector 43(FIG. 8), i.e., a light-emission image to be displayed on the displaysurface of the inner omnidirectional projector 43 (hereinafter alsoreferred to as a display image).

When the users A and B are distant and viewpoints of the users A and Bare therefore distant from each other, for example, as shown in FIG. 21,even if the inner dome screen 44 presents the users A and B with imagesof the object as viewed from their respective viewpoints (viewpointimages), the user A is allowed to view only the viewpoint imageassociated with the viewpoint of the user A while the user B is allowedto view only the viewpoint image associated with the viewpoint of theuser B.

That is, the viewpoint image associated with the viewpoint of the user Bis not seen by the user A, or is seen only to an extent such thatviewing of the viewpoint image associated with the viewpoint of the userA is not interfered. Similarly, the viewpoint image associated with theuser A is not seen by the user B, or is seen only to an extent such thatviewing of the viewpoint image associated with the viewpoint of the userB is not interfered.

On the other hand, when the users A and B are close to each other, forexample, as shown in FIG. 22, their viewpoints are also close to eachother, so that viewpoint images displayed on the inner dome screen 44respectively for the users A and B could overlap each other. Even if theviewpoint images displayed on the inner dome screen 44 respectively forthe users A and B do not overlap each other, the viewpoint images forthe users A and B are displayed in proximity to each other on the innerdome screen 44. Thus, the viewpoint image associated with the viewpointof the user B interferes with viewing by the user A of the viewpointimage associated with the viewpoint of the user A, and the viewpointimage associated with the viewpoint of the user A interferes withviewing by the user B of the viewpoint image associated with theviewpoint of the user B.

Thus, when the distance between the viewpoints of the users A and B isshort, the surrounding-image processing unit 35 sets a common viewpointfor the users A and B at a position close to both of the viewpoints ofthe users A and B, and generates an image of the object as viewed fromthat viewpoint (viewpoint image). In that case, as shown in FIG. 22, theinner dome screen 44 presents both of the users A and B with an image ofthe object as viewed from the common viewpoint, so that image viewing bythe users A and B is not interfered with.

FIG. 23 shows an example configuration of the surrounding-imageprocessing unit 35 shown in FIG. 6, which executes the processingdescribed above (surrounding-image processing). A viewpoint buffer 61receives a viewpoint of a user, output from the viewpoint detecting unit37 (FIG. 6). The viewpoint buffer 61 stores the viewpoint supplied fromthe viewpoint detecting unit 37. If a plurality of users is present, theviewpoint detecting unit 37 shown in FIG. 6 detects viewpoints of theplurality of users, and supplies the viewpoints to the viewpoint buffer61. In that case, the viewpoint buffer 61 stores all the viewpoints ofthe plurality of users, supplied from the viewpoint detecting unit 37.

A viewpoint processing unit 62 reads the viewpoint of the user, storedin the viewpoint buffer 61, executes predetermined processing on theviewpoint as required, and supplies the result to a viewpoint storingunit 63. The viewpoint storing unit 63 stores the viewpoint suppliedfrom the viewpoint processing unit 62. A viewpoint-direction calculatingunit 64 calculates a direction of the viewpoint stored in the viewpointstoring unit 63 (viewpoint direction), and supplies it to aviewpoint-image converting unit 67.

A frame memory 65 stores, on a frame-by-frame basis, surrounding-imagedata supplied from the demultiplexer 32 (FIG. 6), and reads thesurrounding-image data and supplies it to an image selecting unit 66.The frame memory 65 is constructed similarly to the frame memory 51shown in FIG. 14, so that the frame memory 65 allows simultaneousstoring and reading of surrounding-image data.

The image selecting unit 66 selects image data needed to generate aviewpoint image associated with the viewpoint stored in the viewpointstoring unit 63 from the surrounding-image data supplied from the framememory 65. More specifically, the surrounding-image data is a set ofimage data captured by the plurality of surrounding cameras 22 ₁, 22 ₂,22 ₃, . . . disposed at a plurality of camera positions, as shown inFIG. 4, and letting the image data constituting the surrounding-imagedata, captured by a surrounding camera 22 _(k), be referred to ascamera-image data, the image selecting unit 66 selects camera-image datafrom the surrounding-image data based on the viewpoint stored in theviewpoint storing unit 63.

Then, the image selecting unit 66 supplies the camera-image dataselected from the surrounding-image data based on the viewpoint storedin the viewpoint storing unit 63, as it is or after processing it, tothe viewpoint-image converting unit 67 as viewpoint-image dataassociated with the viewpoint.

The viewpoint-image converting unit 67, based on the viewpoint directionsupplied from the viewpoint-direction calculating unit 64 and based onthe shape of the inner dome screen 44 (FIG. 8), generates display-imagedata for displaying an image on the display surface of the inneromnidirectional projector 43 (FIG. 8) using the viewpoint image datasupplied from the image selecting unit 66, and supplies it to a framememory 68.

The image selecting unit 66 and the viewpoint-image converting unit 67described above constitute a display-image generating unit 70 forgenerating display-image data from surrounding-image data based on theviewpoint of the user and the shape of the inner dome screen 44 (FIG.8).

The frame memory 68 stores the display-image data supplied from theviewpoint-image converting unit 67, and supplies the display-image datato the inner omnidirectional projector 43 (FIG. 8). The frame memory 68is constructed similarly to the frame memory 51 shown in FIG. 14, sothat the frame memory 68 allows simultaneous storing and reading ofdisplay-image data.

A controller 69 controls the viewpoint processing unit 62, theviewpoint-direction calculating unit 64, the image selecting unit 66,and the viewpoint-image converting unit 67 with reference to the statusof storage of surrounding-image data in the frame memory 65.

Next, processing that is executed by the viewpoint-direction calculatingunit 64 shown in FIG. 23 will be described with reference to FIG. 24.The viewpoint-direction calculating unit 64 calculates a longitude and alatitude that represent a direction of the viewpoint stored in theviewpoint storing unit 63 (viewpoint direction).

Now, letting a three-dimensional coordinate system in which the opticalaxis of the inner omnidirectional projector 43 constitutes a z axis andin which the display surface of the inner omnidirectional projector 43constitutes an xy plane be referred to as a surrounding-image-displaycoordinate system, the viewpoint-direction calculating unit 64calculates, as a viewpoint direction, the direction of a viewpoint V(x,y, z) at a certain position as viewed from the origin O in thesurrounding-image-display coordinate system.

Now, with regard to the surrounding-image-display coordinate system,letting an angle with respect to the x axis on the xy plane be referredto as a longitude θ, an angle with respect to the z axis be referred toas a latitude φ, and the direction of a point with a longitude θ and alatitude φ be denoted as a direction (θ, φ), the longitude θ and thelatitude φ representing the viewpoint direction (θ, φ) of the viewpointV(x, y, z) can be calculated by the following equations, respectively.

$\begin{matrix}{\theta = {\tan^{- 1}\frac{y}{x}}} & (6) \\{\phi = {\frac{\pi}{2} - {\tan^{- 1}\frac{z}{\sqrt{x^{2} + y^{2}}}}}} & (7)\end{matrix}$

The viewpoint-direction calculating unit 64 calculates the viewpointdirection (θ, φ) according to equations (6) and (7), and supplies it tothe viewpoint-image converting unit 67.

Next, processing that is executed by the display-image generating unit70 shown in FIG. 23 will be described with reference to FIGS. 25 to 27.

In order to present a user with a display on the inner dome screen 44(FIG. 8) of a viewpoint image of an object as viewed from a viewpoint ofthe user, simply, for example, as shown in FIG. 25, camera-image datacaptured from a viewpoint direction (θ, φ) of the user's viewpoint mustbe displayed as a viewpoint image in a predetermined range on the innerdome screen 44, centered at an intersection U of the viewpoint direction(θ, φ) and the inner dome screen 44.

Now, letting the viewpoint image, i.e., the camera-image data capturedfrom the viewpoint direction (θ, φ), be denoted as a viewpoint image (θ,φ), and the predetermined range on the inner dome screen 44, centered atthe intersection U of the viewpoint direction (θ, φ) of the viewpoint ofthe user and the inner dome screen 44 as a display area (θ, φ, in orderto display the viewpoint image (θ, φ) in the display area (θ, φ), theviewpoint image (θ, φ) must be written to a position on a display imageto be displayed on the display surface of the inner omnidirectionalprojector 43 (FIG. 8).

Thus, in the display-image generating unit 70 shown in FIG. 23, theimage selecting unit 66 selects the viewpoint image (θ, φ), i.e., thecamera-image data captured from the viewpoint direction (θ, φ) of theviewpoint stored in the viewpoint storing unit 63, and theviewpoint-image converting unit 67 maps the viewpoint image (θ, φ) to adisplay image so that the viewpoint image (θ, φ) can be displayed in thedisplay area (θ, φ)

Now, let a two-dimensional coordinate system in which the displaysurface of the inner omnidirectional projector 43 constitutes an xyplane be referred to as a display-surface coordinate system, and an xcoordinate and a y coordinate thereof as an x_(a) coordinate and a y_(a)coordinate, as shown in FIG. 26. Furthermore, let a two-dimensionalcoordinate system in which the viewpoint image (θ, φ) constitutes an xyplane be referred to as a viewpoint-image coordinate system. As shown inFIG. 26, the origin O_(a) of the display-surface coordinate system istaken at an intersection of the optical axis of the inneromnidirectional projector 43 and the display surface, and the origin Oof the viewpoint-image coordinate system is taken at a center of therectangular viewpoint image (θ, φ) (intersection of diagonals of therectangular viewpoint image (θ, φ))

In order to display the viewpoint image (θ, φ) in the display area (θ,φ), first, the viewpoint image (θ, φ) must be mapped to a position thatis distant from the origin O_(a) of the display-surface coordinatesystem by a distance g(φ) corresponding to the latitude φ, as shown inFIG. 26. Furthermore, in order to display the viewpoint image (θ, φ) inthe display area (θ, φ), second, the viewpoint image (θ, φ) must berotated by an angle of (π/2+θ) counterclockwise, as shown in FIG. 27.

Thus, the viewpoint-image converting unit 67 first calculates, accordingto equation (8), coordinates (x_(ao), y_(ao)) in the display-surfacecoordinate system of the origin O of the viewpoint-image coordinatesystem of the viewpoint image (θ, φ)

$\begin{matrix}{\begin{pmatrix}x_{SO} \\y_{SO}\end{pmatrix} = \begin{pmatrix}{{g(\phi)}\cos\;\theta} \\{{g(\phi)}\sin\;\theta}\end{pmatrix}} & (8)\end{matrix}$

In equation (8), the function g(φ) represents the opticalcharacteristics of the inner omnidirectional projector 43 (FIG. 8) (afunction that is determined by a projection method of the fish-eye lensof the inner omnidirectional projector 43).

Furthermore, letting coordinates in the viewpoint-image coordinatesystem of a pixel Q constituting the viewpoint image (θ, φ) be denotedas Q(x_(Q), y_(Q)), and coordinates thereof in the display-surfacecoordinate system as Q(x_(aQ), y_(aQ)), the viewpoint-image convertingunit 67 calculates, according to equation (9), coordinates Q(x_(aQ),y_(aQ)) in the display-surface coordinate system of the pixel Qconstituting the viewpoint image (θ, φ).

$\begin{matrix}{\begin{pmatrix}x_{SQ} \\y_{SQ}\end{pmatrix} = {{\begin{pmatrix}{\cos\left( {\frac{\pi}{2} + \theta} \right)} & {- {\sin\left( {\frac{\pi}{2} + \theta} \right)}} \\{\sin\left( {\frac{\pi}{2} + \theta} \right)} & {\cos\left( {\frac{\pi}{2} + \theta} \right)}\end{pmatrix}\begin{pmatrix}x_{Q} \\y_{Q}\end{pmatrix}} + \begin{pmatrix}x_{S0} \\y_{S0}\end{pmatrix}}} & (9)\end{matrix}$

Then, the viewpoint-image converting unit 67 writes the pixel value ofeach pixel Q(x_(Q), y_(Q)) constituting the viewpoint image (θ, φ) inthe frame memory 68, at a position corresponding to coordinates (x_(aQ),y_(aQ)) expressed by equation (9), thereby converting the viewpointimage (θ, φ) into a display image.

Next, processing that is executed by the surrounding-image processingunit 35 shown in FIG. 23 (surrounding-image processing) will bedescribed with reference to flowcharts shown in FIGS. 28 and 29.

The controller 69 monitors the frame memory 65, and it startssurrounding-image processing when the frame memory 65 starts storingsurrounding-image data.

More specifically, in the surrounding-image processing, first in stepS21, the controller 69 sets an initial value, for example, 1, in avariable i for counting the number of frames, and the processing thenproceeds to step S22.

The process of setting 1 in the variable i in step S21, and a process ofincrementing the variable i in step S53, which will be described later,are executed respectively in synchronization with the process of setting1 in the variable i in step S11 and the process of incrementing thevariable i in step S11, described earlier and shown in FIG. 18. That is,the wide-angle-image processing in the wide-angle-image processing unit33 and the surrounding-image processing in the surrounding-imageprocessing unit 35 are executed in synchronization on the same frame.Thus, the wide-angle-image display unit 34 and the surrounding-imagedisplay unit 36 display an immersion image and a bird's-eye image of thesame frame in synchronization.

In step S22, the controller 69 sets an initial value, for example, 1, ina variable n for counting the number of users, and the processing thenproceeds to step S23. In step S23, the viewpoint processing unit 62reads the viewpoint of a user #n from the viewpoint buffer 61, andregisters it in the viewpoint storing unit 63.

More specifically, the viewpoint detecting unit 37 (FIG. 6) detectsviewpoints of all the users present inside the outer dome screen 42(FIG. 7), for example, at a frame cycle, and updates contents stored inthe viewpoint buffer 61 with the latest viewpoints detected. In stepS23, the viewpoint processing unit 62 reads the viewpoint of the user#n, stored in the viewpoint buffer 61 at the timing of an i-th frame,and registers it in the viewpoint storing unit 63. The viewpoint of theuser #n may be stored in the viewpoint storing unit 63 together with,for example, an ID (identification) representing the user #n.

After the viewpoint processing unit 62 has registered the viewpoint ofthe user #n in the viewpoint storing unit 63, the processing proceeds tostep S24, in which the controller 69 determines whether the variable nequals the total number N of users present inside the outer dome screen42 (FIG. 7). If it is determined in step S23 that the variable n doesnot equal N, the processing proceeds to step S25, in which thecontroller 69 increments the variable n by 1. The processing thenreturns to step S23, and the same processing is repeated.

On the other hand, if it is determined in step S24 that the variable nequals N, that is, when viewpoints of all the users present inside theouter dome screen 42 (FIG. 7) have been stored in the viewpoint storingunit 63, the processing proceeds to step S26. In step S26, thecontroller 69 sets, as an initial value, the total number N of userspresent inside the outer dome screen 42 (FIG. 7) in a variable Mrepresenting the total number of viewpoints to be processed in theviewpoint selecting unit 66. The processing then proceeds to step S27.

In step S27, the controller 69 sets an initial value, for example, 1, ina variable m for counting the number of viewpoints, and sets an initialvalue in a flag indicating the presence or absence of short-rangeviewpoints, for example, 0 indicating the absence of short-rangeviewpoints, which will be described later. The processing then proceedsto step S28.

In step S28, the viewpoint processing unit 62, with regard to an m-thviewpoint among the M viewpoints stored in the viewpoint storing unit63, calculates distances from the other (M−1) respective viewpoints(viewpoint distances), and detects viewpoints with viewpoint distancesnot greater than (or distances smaller than) a predetermined thresholdvalue TH1 as short-range viewpoints for the m-th viewpoint. Theprocessing then proceeds to step S29, in which the viewpoint processingunit 62 determines whether a short-range viewpoint exists for the m-thviewpoint.

If it is determined in step S29 that a short-range viewpoint does notexist for the m-th viewpoint, that is, if a viewpoint with a viewpointdistance from the m-th viewpoint not greater than the threshold valueTH1 is not stored (registered) in the viewpoint storing unit 63, stepsS30 to S34 are skipped and the processing proceeds to step S35.

On the other hand, if it is determined in step S29 that a short-rangeviewpoint exists for the m-th viewpoint, that is, if a viewpoint with aviewpoint distance from the m-th viewpoint not greater than thethreshold value TH1 (short-range viewpoint) is stored (registered) inthe viewpoint storing unit 63, the processing proceeds to step S30. Instep S30, the controller 69 sets, for example, 1 in the flag, indicatingthe presence of a short-range viewpoints, and the processing thenproceeds to step S31.

The process in step S30 can be skipped if 1 is already set in the flag.

In step S31, the viewpoint processing unit 62 detects the number ofviewpoints determined as short-range viewpoints for the m-th viewpointamong the viewpoints stored in the viewpoint storing unit 63, andfurther detects a barycenter (a center of mass between two associatedobjects) of the short-range viewpoints and the m-th viewpoint.

Then, the processing proceeds to step S32, in which the viewpointprocessing unit 62 registers (writes) the barycenter calculated in stepS31 as the m-th viewpoint in the viewpoint storing unit 63 byoverwriting, and the processing then proceeds to step S33. In step S33,the viewpoint processing unit 62 deletes the viewpoints determined asshort-range viewpoints for the m-th viewpoint among the viewpointsstored in the viewpoint storing unit 63, and the processing thenproceeds to step S34.

In step S34, the controller 69 subtracts the number of short-rangeviewpoints detected in step S31 from the variable M representing thetotal number M of viewpoints to be processed in the viewpoint selectingunit 66, and newly sets the resulting value of subtraction to thevariable M, and the processing then proceeds to step S35. That is, sincethe short-range viewpoints among the viewpoints stored in the viewpointstoring unit 63 are deleted in step S33, the value of the variable M isdecreased in step S34 by the number of short-range viewpoints deleted.

Thus, by the processing from step S28 to step S34, enclosed by a dottedline in FIG. 28, one or more viewpoints that are close in distance tothe m-th viewpoint are detected, and the one or more viewpoints and them-th viewpoint are integrated into a position at the barycenter of theseviewpoints.

In step S35, the controller 69 determines whether the variable m equalsthe variable M, i.e., the number of viewpoints M stored in the viewpointstoring unit 63. If it is determined in step S35 that the variable mdoes not equal the variable M, the processing proceeds to step S36, inwhich the controller 69 increments the variable m by 1. The processingthen returns to step S28, and the same processing is repeated.

On the other hand, if it is determined in step S35 that the variable mequals the variable M, the processing proceeds to step S37, in which thecontroller 69 determines whether the flag equals 1. If it is determinedin step S37 that the flag equals 1, that is, if a viewpoint having beenintegrated in the previous loop processing from step S28 to step S36exists among the viewpoints stored in the viewpoint storing unit 63 andif viewpoints could be further integrated, the processing returns tostep S27, and the same processing is repeated.

On the other hand, if it is determined in step S37 that the flag doesnot equal 1, that is, if the flag remains 0 as set in step S27 and if aviewpoint with a distance from other viewpoints not greater than thethreshold value TH1 does not exist among the viewpoints stored in theviewpoint storing unit 63, the processing proceeds to step S41 shown inFIG. 29.

In step S41, the controller 69 sets an initial value, for example, 1, tothe variable m for counting the number of viewpoints, and the processingproceeds to step S42. In step S42, the image selecting unit 66calculates, as camera distances, distances between the m-th viewpointstored in the viewpoint storing unit 66 and the camera viewpoints of thesurrounding cameras 22 _(k) constituting the surrounding imaging unit12.

The camera viewpoints of the surrounding cameras 22 _(k) refer to camerapositions that are multiplexed with surrounding-image data, normalized(divided) by a predetermined normalization factor β. The normalizationfactor β may be, for example, a value obtained by dividing a distancefrom the origin to a most remote viewpoint that a user can take in thesurrounding-image-display coordinate system by a distance from an objectto a surrounding camera 22 _(k) that is farthest from the object.

After the process of step S42, the processing proceeds to step S43, inwhich the image selecting unit 66 calculates, as a minimum cameradistance, a minimum value D_(min) among the camera distances calculatedfor the m-th viewpoint in step S42, and the processing proceeds to stepS44. In step S44, the image selecting unit 66 determines whether theminimum camera distance D_(min) for the m-th viewpoint is not greaterthan (or is smaller than) a predetermined threshold value TH2.

If it is determined in step S44 that the minimum camera distance D_(min)for the m-th viewpoint is not greater than the predetermined thresholdvalue TH2, that is, if the camera viewpoint with the minimum cameradistance D_(min) is close to the m-th viewpoint and can therefore beconsidered as the m-th viewpoint, the processing proceeds to step S45.In step S45, the image selecting unit 66 selects, as a viewpoint imagefor the m-th viewpoint, an i-th frame camera image captured by asurrounding camera 22 _(k) having a camera viewpoint with the minimumcamera distance D_(min), from among the surrounding image data stored inthe frame memory 65, and supplies it to the viewpoint-image convertingunit 67. The processing then proceeds to step S49.

On the other hand, if it is determined in step S44 that the minimumcamera distance D_(min) for the m-th viewpoint is greater than thepredetermined threshold value TH2, that is, if the camera viewpoint withthe minimum camera distance D_(min) is not close to the m-th viewpointand therefore cannot be considered as the m-th viewpoint, the processingproceeds to step S47. In step S47, the image selecting unit 66 detects,as a second camera distance D₂, the second smallest camera distancecalculated for the m-th viewpoint, and selects two camera images fromamong the surrounding-image data stored in the frame memory 65, namely,an i-th frame camera image captured by a surrounding camera 22 _(k)having a camera viewpoint with the minimum camera distance D_(min) andan i-th frame camera image captured by a surrounding camera 22 _(k)having a camera viewpoint with the second camera distance D₂.Furthermore, in step S47, the image selecting unit 66 combines the i-thframe camera image captured by the surrounding camera 22 _(k) having thecamera viewpoint with the minimum camera distance D_(min) and the i-thframe camera image captured by the surrounding camera 22 _(k) having thecamera viewpoint with the second camera distance D₂ by a ratioD₂:D_(min) of the second camera distance D₂ and the minimum cameradistance D_(min), thereby generating a synthesized image.

That is, letting a c-th pixel constituting the i-th frame camera imagecaptured by the surrounding camera 22 _(k) having the camera viewpointwith the minimum camera distance D_(min) be denoted as p_(c), and a c-thpixel constituting the i-th frame camera image captured by thesurrounding camera 22 _(k) having the camera viewpoint with the secondcamera distance D₂ be denoted as q_(c), the image selecting unit 66generates a synthesized pixel by combining the pixels p_(c) and q_(c)according to a formula (D₂×p_(c)+D_(min)×q_(c))/(D₂+D_(min)), andgenerates a synthesized image composed of such synthesized pixels.

Although a synthesized image is generated using two camera images in theexample given above, alternatively, a synthesized image may be generatedusing three or more camera images.

The processing then proceeds to step S48, in which the image selectingunit 66 selects the synthesized image generated in step S47 as aviewpoint image for the m-th viewpoint, and supplies it to theviewpoint-image converting unit 67. The processing then proceeds to stepS49.

In step S49, the viewpoint image for the m-th viewpoint is convertedinto a display image. More specifically, in step S49, theviewpoint-direction calculating unit 64 calculates a viewpoint direction(θ, φ) of the m-th viewpoint stored in the viewpoint storing unit 63,and supplies it to the viewpoint-image converting unit 67. Theviewpoint-image converting unit 67, based on the viewpoint direction (θ,φ) of the m-th viewpoint, supplied from the viewpoint-directioncalculating unit 64, calculates a position of a display image where eachpixel of the viewpoint image for the m-th viewpoint, supplied from theimage selecting unit 66, is to be mapped, according to equations (8) and(9). Then, the viewpoint-image converting unit 67 maps (writes) eachpixel of the viewpoint image for the m-th viewpoint to a position(address) on the frame memory 68 corresponding to the position of thedisplay image thus calculated. The processing then proceeds to step S50.

Then, light beams corresponding to the display image written to theframe memory 68 are emitted from the inner omnidirectional projector 43,whereby the inner dome screen 44 displays an image of the object asviewed from the m-th viewpoint.

In step S50, the controller 69 determines whether the variable m equalsthe total number M of the viewpoints stored in the viewpoint storingunit 63. If it is determined in step S50 that the variable m does notequal M, the processing proceeds to step S51, in which the controller 69increments the variable m by 1. The processing then returns to step S42,and the same processing is repeated.

On the other hand, if it is determined in step S50 that the variable mequals M, that is, if viewpoint images for all the viewpoints stored inthe viewpoint storing unit 63 have been converted into display imagesand the display images have been written to the frame memory 68, theinner omnidirectional projector 43 (FIG. 8) emits light beamscorresponding to the display images stored in the frame memory 68 asimages of the i-th frame. Accordingly, the inner dome screen 44 displaysbird's-eye images of the objects as viewed respectively from the firstto M-th viewpoints at positions in accordance with the respectiveviewpoints.

Then, the processing proceeds to step S52, in which the controller 69refers to the frame memory 65 to determine whether the i-th frame is thelast frame. If it is determined in step S52 that the i-th frame is notthe last frame, that is, if surrounding-image data of a frame subsequentto the i-th frame is stored in the frame memory 65, the processingproceeds to step S53. In step S53, the controller 69 increments thevariable i by 1. The processing then returns to step S22 shown in FIG.28, and the same processing is repeated.

On the other hand, if it is determined in step S52 that the i-th frameis the last frame, that is, if surrounding-image data of a framesubsequent to the i-th frame is not stored in the frame memory 65, thesurrounding-image processing is exited.

As described above, in the display system 2, an immersion image of thei-th frame is displayed on the inner surface of the outer dome screen42, and a bird's-eye image of the i-th frame is displayed on the outersurface of the inner dome screen 44. Accordingly, the user is allowed toexperience a sense of immersion provided by the immersion image, and isalso allowed to readily grasp his/her own position by the bird's-eyeimage.

Thus, for example, if the imaging/display system shown in FIG. 1 is usedfor imaging and display of a sports program, a large number ofsurrounding cameras 22 _(k) (FIG. 4) of the imaging system 1 is disposedsuccessively so as to surround a stadium where the sports event is beingheld, and the omnidirectional camera 21 (FIG. 3) is disposed at a centerof the stadium so as to allow imaging. Accordingly, the display systemis allowed to display an image that provides a sense of presence.

Furthermore, the inner dome screen 44 displays a bird's-eye image inaccordance with a viewpoint of a user at a position in accordance withthe viewpoint of the user, so that the bird's-eye image presented to theuser changes as the user moves his/her viewpoint. Thus, the user isallowed to “look into” an object, that is, moves his/her viewpoint toview an object that has been hidden as viewed from a certain viewpoint.Furthermore, bird's-eye images of an object as viewed from respectiveviewpoints of a plurality of users can be presented to the plurality ofusers.

Furthermore, with the display system 2, for example, the outer domescreen 42 (FIG. 7) displays fish, animals, etc. that are remote whilethe inner dome screen 44 (FIG. 8) displays fish, animals, etc. that arenearby, thereby virtually implementing a nature observing facility suchas an aquarium or zoo.

Furthermore, the display system 2 allows display of an artificiallyproduced image such as computer graphics instead of an image actuallycaptured. That is, the display system 2 allows display of a computergraphic image of the cosmic space, microscopic structure of molecules,etc. In that case, an interface that allows a user to intuitively graspa structure that cannot usually be recognized by the vision is provided.

Furthermore, the display system 2 can be used, for example, fordisplaying images in what is called a video game.

The series of processing steps by the wide-angle-image processing unit33 and the surrounding-image processing unit 35 shown in FIG. 6 may beexecuted either by hardware or by software. When the series ofprocessing steps is executed by software, programs constituting thesoftware are installed, for example, on a general-purpose computer.

FIG. 30 shows an example configuration of a computer on which theprograms for executing the series of processing steps described aboveare to be installed, according to an embodiment.

The programs can be stored in advance in a hard disc 105 or a ROM 103,which is a recording medium included in the computer.

Alternatively, the programs may be temporarily or permanently stored(recorded) on a removable recording medium 111 such as a flexible disc,a CD-ROM (compact disc read only memory), an MO (magneto optical) disc,a DVD (digital versatile disc), a magnetic disc, or a semiconductormemory. The removable recording medium 111 can be provided as so-calledpackage software.

Instead of installing the programs onto the computer from the recordingmedium 111 as described above, the programs may be transferred bywireless from a downloading site to the computer via an artificialsatellite for digital satellite broadcasting, or transferred throughwires via networks such as a LAN (local area network) and the Internetso that the programs transferred to the computer will be received by acommunication unit 108 and can be installed onto the internal hard disc105.

The computer includes a CPU (central processing unit) 102. The CPU 102is connected to an input/output interface 110 via a bus 101. When theCPU 102 receives input of a command issued by a user by operating aninput unit 107 including, for example, a keyboard, a mouse, and amicrophone, the CPU 102 executes a program stored in the ROM (read onlymemory) 103 according to the command. Alternatively, the CPU 102 loadsinto a RAM (random access memory) and executes a program stored in thehard disc 105, a program transferred via a satellite or a network,received by the communication unit 108, and installed on the hard disc105, or a program read from the removable recording medium 111 andinstalled on the hard disc 105. The CPU 102 thus executes theabove-described processing according to the flowcharts or processingexecuted with the above-described arrangements shown in the blockdiagrams. The CPU 102 then outputs the processing result from an outputunit 106 including, for example, an LCD (liquid crystal display) and aspeaker, sends it from the communication unit 108, or records it on thehard disc 105 as required, for example, via the input/output interface110.

The processing steps defining the programs for allowing the computer toexecute the various processes in this specification need not necessarilybe executed sequentially in the time order shown in the flowcharts, andmay be executed in parallel or individually (e.g., parallel processingor processing using objects).

Furthermore, the programs may be executed either by a single computer orby distributed processing with a plurality of computers. Furthermore,the programs may be transferred to and executed by a remote computer.

In the display system 2, an image recorded on the recording medium 3(FIG. 1) can be displayed in an off-line manner, and also, an imagetransmitted via the transmission medium 4 can be displayed in real time.

Furthermore, as described earlier, the inner omnidirectional projector43 (FIG. 8) emits light beams corresponding to a display image, and thelight beams are back-projected onto the inner dome screen 44 (FIG. 8);thus, when a viewpoint image is displayed, the left and right of theviewpoint image is reversed. In order to avoid the left-right reversalof the viewpoint image displayed on the inner dome screen 44, forexample, the left and right of a camera image that serves as an originalfor the viewpoint image must be reversed in advance. This reversal canbe executed, for example, by the image selecting unit 66 (FIG. 23) ofthe surrounding-image processing unit 35. Alternatively, the left-rightreversal of the camera image may be executed optically when the cameraimage is captured by a surrounding camera 22 _(k) (FIG. 4) of theimaging system 1.

Furthermore, in this embodiment, the inner omnidirectional projector 43(FIG. 8) emits light beams corresponding to a display image, and thelight beams are back-projected to the inner dome screen 44 (FIG. 8),whereby a viewpoint image is displayed. In this case, the viewpointimage must be generated with consideration of the shape of the innerdome screen 44 (FIG. 8). More specifically, considering that theviewpoint image is rectangular, when the rectangular viewpoint image ismapped to a particular position of the display image, light beamscorresponding to the display image are emitted by the inneromnidirectional projector 43, whereby the viewpoint image is displayedon the inner dome screen 44, since the rectangular viewpoint image isprojected onto the spherical surface of the inner dome screen 44 havinga shape of a hemisphere, the viewpoint image displayed on the inner domescreen 44 is distorted. Thus, in order that the inner dome screen 44display the viewpoint image in a rectangular shape without distortion,the viewpoint-image converting unit 67 (FIG. 23) must convert theviewpoint image and map it to a display image based on the shape of theinner dome screen 44.

In this embodiment, however, the radius of the inner dome screen 44(FIG. 8) is large to a certain extent, and the range where the viewpointimage is displayed can be considered as planar. In this case, theconversion and mapping of the viewpoint image to a display image basedon the shape of the inner dome screen 44 is optional.

Furthermore, although only one surrounding imaging unit 12 (FIG. 2) isdisposed in the imaging system 1 in this embodiment, a plurality ofsurrounding imaging units may be provided. In that case, the arrangementmay be such that the plurality of surrounding imaging units capturesimages of a plurality of objects that are spatially or temporallyseparated, and that the display system 2 processes an image obtained bysuperimposing the images of the plurality of objects as a camera image.

Furthermore, the wide-angle-image processing unit 33, thesurrounding-image processing unit 35, and the viewpoint detecting unit37 shown in FIG. 6 may be provided in the imaging system 1 (FIG. 2)instead of the display system 1.

The present patent specification contains subject matter related to thatdisclosed in Japanese priority document, application numberP2002-291819, filed in the Japanese patent office on Oct. 4, 2002, theentire contents of which being incorporated herein by reference.

For convenience, the following reference numerals are used throughoutthe figures to represent the various components employed in one or moreembodiments according to the present invention. 1: imaging system; 2:display system; 3: recording medium; 4: transmission medium; 11:wide-angle imaging unit; 12: surrounding imaging unit; 13: multiplexer;14: recording/sending unit; 21: omnidirectional camera; 22 _(k):surrounding cameras; 31: playback/receiving unit; 32: demultiplexer; 33:wide-angle-image processing unit; 34: wide-angle-image display unit; 35:surrounding-image processing unit; 36: surrounding-image display unit;37: viewpoint detecting unit; 41: outer omnidirectional projector; 42:outer dome screen; 43: inner omnidirectional projector; 44: inner domescreen; 51: frame memory; 52: wide-angle-image converting unit; 53:image-angle correcting unit; 54: latitude-longitude-image convertingunit; 55: frame memory; 56: controller; 61: viewpoint buffer; 62:viewpoint processing unit; 63: viewpoint storing unit; 64:viewpoint-direction calculating unit; 65: frame memory; 66: imageselecting unit; 67: viewpoint-image converting unit; 68: frame memory;69: controller; 70: display-image generating unit; 101: bus; 102: CPU;103: ROM; 104: RAM; 105: hard disc; 106: output unit; 107: input unit;108: communication unit; 109: drive; 110: input/output interface; 111:removable recording medium

1. A display apparatus, comprising: a first display structure having a3-dimensional shape with an interior surface, said first displaystructure being configured to display a first image on said interiorsurface so that the first image is viewable from a position that is atleast partially surrounded by said interior surface; and a seconddisplay structure, separate from the first display structure, configuredto display a second image on a surface of the second display structureso that the second image is simultaneously viewable from said position.2. A display apparatus according to claim 1, wherein the first displaystructure is configured to display an immersion image on said interiorsurface so as to present an user with a sense of immersion.
 3. A displayapparatus according to claim 1, further comprising: a projectorconfigured to project the first image toward the interior surface.
 4. Adisplay apparatus according to claim 3, wherein the first imageprojected by the projector is an omnidirectional image captured by anomnidirectional imaging device configured to image surroundings insubstantially all directions visible from the omnidirectional imagingdevice.
 5. A display apparatus according to claim 4, further comprising:the omnidirectional imaging device.
 6. A display apparatus according toclaim 4, further comprising: an omnidirectional-image convertingmechanism configured to convert the omnidirectional image into alatitude-longitude image having a rectangular shape on a plane definedby a latitudinal direction and a longitudinal direction; a correctionmechanism configured to correct the latitude-longitude image based oncharacteristics of the projector; and a latitude-longitude-imageconverting mechanism configured to convert the latitude-longitude imageafter having been corrected by the correction mechanism into theomnidirectional image; wherein the projector is configured to projectthe omnidirectional image obtained by the latitude-longitude-imageconverting mechanism.
 7. A display apparatus according to claim 1,wherein the second display structure is configured to display abird's-eye image as viewed from a predetermined viewpoint within said3-dimensional shape.
 8. A display apparatus according to claim 1,further comprising: a projector configured to project the second imagefrom within an interior space that is defined by an inside of the seconddisplay structure toward the surface.
 9. A display apparatus accordingto claim 8, wherein said second image is an object, and the projector isconfigured to project said second image to the surface for viewing froma predetermined viewpoint.
 10. A display apparatus according to claim 9,wherein the projector is configured to project said second image to thesurface that is viewable from a viewpoint of an user.
 11. A displayapparatus according to claim 10, further comprising: a viewpointdetector configured to detect the viewpoint of the user.
 12. A displayapparatus according to claim 9, wherein the projector is configured toproject said second image based on the predetermined viewpoint and ashape of the surface on which the second image is displayed.
 13. Adisplay apparatus according to claim 1, wherein at least one of thefirst display structure and the second display structure includes ahemispherical dome.